HISP: Heterogeneous Image Signal Processor Pipeline Combining Traditional and Deep Learning Algorithms Implemented on FPGA

To tackle the challenges of edge image processing scenarios, we have developed a novel heterogeneous signal processor (HISP) pipeline combining advantages traditional processors and deep learning ISP (DLISP). Through multi-dimensional quality assessment (IQA) system integrating methods like RankIQA, BRISQUE, SSIM, various partitioning schemes were compared to explore highest-quality imaging scheme. The UNet-specific deep-learning unit (DPU) based on field programmable gate array (FPGA) provided 14.67× acceleration ratio for total network deconvolution max pool, calculation latency was as low 2.46 ms 97.10 ms, achieving an impressive speedup 46.30× 36.49× with only 4.04 W power consumption. HISP consisting DPU FPGA-implemented (ISP) submodules, which scored highly in system, single time 524.93 consumption 8.56 W, low-cost fully replicable solution extremely illumination high noise environments.